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Introduction 


The Program for International Student Assessment (PISA) conducted by 
the OECD in three-year cycles is an international large-scale program that 
measures 15-year-old students’ knowledge and skills in reading, mathemat- 
ics, and science. PISA compares the quality and equity of education across 
countries, and the results obtained are used by educators and politicians in 
determining appropriate education policies and practices (OECD, 2019a). 
In each cycle of the research, a subject is chosen as the main area. Like the 
previous cycles, the PISA 2018 assessment covered the areas of reading, 
mathematics, and science, but the core area was reading literacy. 

In recent years resources related to ICT have become accessible to stu- 
dents both at home and at school and have gained a large place in modern 
society. Due to the importance of ICT, an optional ICT familiarity question- 
naire about ICT availability, ICT skills and ICT use has been included in each 
cycle of PISA since 2000 (Kunina-Habenicht & Goldhammer, 2020). With the 
change in PISA 2015, ICT engagement questionnaire was added to PISA's ICT 
familiarity survey as a new structure and this scale was also included in PISA 
2018. ICT engagement scale has been improved by both intrinsic and extrinsic 
motivation derived from the theory of self-determination (Deci & Ryan, 2000). 
ICT engagement scale has four factors: 1. Interest in ICT which is the intrinsic 
motivation of using products related to ICT (Zylka et al., 2015); 2. Perceived 
ICT competence, students’ ICT-based knowledge and skills that can be used 
to perform ICT-related tasks (Meng at al.,2019); 3. Perceived autonomy in us- 
ing ICT, perceived control and independence of an individual in using ICT; 4. 
Social Relatedness in using ICT, the measure of an individual communicating 
and interacting with others using ICT (Kuger et al., 2016; Zylka et al., 2015) 

ICT engagement scale is a fairly new scale. While reviewing the literature, 
it was noticed that the early studies leading to this scale originated from the 
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a multidimensional construct, plays an 
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The main purpose of this research was 

to explore the effects of ICT engagement 
factors on science performance across 
Singapore and Turkey conditional to the 
sufficient degree of measurement invari- 
ance of ICT engagement scale. The multi- 
group confirmatory factor analysis results 
demonstrated strong factorial invariance 
of ICT engagement scale across Singapore 
and Turkey, so we were able to use ICT 
engagement scale to meaningful and valid 
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countries. While interest in ICT, perceived 
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in using ICT have significant positive direct 
effect on science performance in both 
countries, the direct effect of social related- 
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effects of all ICT engagement constructs on 
student’s performance are higher in Turkey. 
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use of ICT in school and in home (Meng et al., 2019). Researchers obtained mixed results about the effect of using 
computer and technology on students’ success (Agasisti et al., 2020; Chen & Wang, 2013; Eickelmann et al., 2016; 
Lei & Zhao, 2007; Liemb et al.,2014; Notten & Kraaykamp, 2009; Thiessen & Looker, 2007; Xin & Zhou, 2010). For 
example, Thiessen and Looker (2007) showed that intensive computer use in Canada was negative relationship 
with students’ reading performance whereas Xin and Zhou (2010) showed that computer technology use increased 
students’ performance. Using PISA 2012 data from 15 European countries, the study showed that intensive com- 
puter use in homework negatively affected test scores in all areas (Agasisti et al., 2020). It could be argued that the 
first studies about ICT in the literature were about the effect of computer or internet use on success in school and 
in home (Meng et al., 2019). These studies revealed the need for developing an ICT engagement construct. Later 
studies focused on development of construct related to ICT (De Wit et al., 2012; Janneck et al., 2013; Senkbeil & Ihme, 
2017). Zylka et al. (2015) proposed a new ICT engagement scale which was more motivation and meta-cognition 
perspective of ICT related constructs. ICT engagement scale was adapted to the four-dimensional structure in PISA 
2015. In recent studies, ICT engagement has been considered as a multidimensional construct and its effect on 
academic achievement has been explored. Different results were obtained (Cheema & Zhang, 2013; Hu et al., 2018; 
Lee & Wu, 2012; Luu & Freeman, 2011). For example, perceived ICT autonomy had a positive effect on achievement 
(Cheema & Zhang, 2013). According to the study conducted by Lee and Wu in 2012, interest in ICT and perceived 
ICT competence were positive predictors of reading achievement. 

The ICT engagement scale, developed in PISA 2015 and used in PISA 2018, is a relatively new assessment tool. 
There were only two studies (Meng et al., 2019; Ma & Qin, 2021) exploring the validity of this scale across countries. 
It was necessary to explore the measurement invariance of this scale to make valid comparisons between different 
groups (Mata et al., 2020). Measurement invariance is a property that indicates a measurement tool (a question- 
naire in the case of survey research) measures the same concept in various subgroups in the same way (Davidov et 
al., 2014). The measurement issues of the newly ICT engagement scale need to be further explored. This research 
was devoted to explore the effects of ICT engagement factors on performance of science between Singapore and 
Turkey using Multi-group Structural Equation Modeling (Multi-group SEM) analysis. The first step was composed 
of the analyzes for configural, metric and scalar invariance of ICT engagement scale of PISA 2018 across Singapore 
and Turkey using Multi-group Confirmatory Factor Analysis (Multi-group CFA). As the ICT engagement scale pro- 
vided acceptable invariant, effects of ICT engagement factors on students’ science performance were compared 
between Singapore and Turkey in the second step. 


Research Problem 


In this research, Singapore and Turkey were chosen to compare. Students from these two countries completed 
the optional ICT engagement form in PISA 2018. These two countries represent different education systems and 
different success in PISA. Singapore education system can offer a customized curriculum for student groups created 
according to their technical and social characteristics in the same school, whereas Turkey education system offers 
the same curriculum for all students (Kilic Depren, 2020). Singapore and Turkey had different science performance 
in PISA 2018. According to PISA 2018 results, the average science performance of 15-year-olds in Turkey was 468 
points and it was less than 489 points which was the average for OECD countries participating in the same cycle. 
Turkey's change in performance of science between PISA 2015 and 2018 indicated one of the strongest increases 
among participating countries and economies. In Singapore, whose average in science performance was one 
of the highest among countries and economies participating in PISA, the average score was 551 (OECD, 2019c). 
Singapore has national plans for ICT integration and implementation covering the educational environment and 
resources. Turkey’s ICT policy also aims at improving both ICT infrastructure and capacity of stakeholders according 
to “Information Society and Action Plan (ISAP) for 2014-2018" prepared by The Ministry of Development (MoD) of 
Turkey. Considering that both countries are focused on the same goal in terms of ICT, Singapore, with the highest 
science achievement, and Turkey, with the highest increase in science performance between PISA 2015 and 2018, 
were selected to compare the effectiveness of ICT policy. 


Research Focus 


In this research, the measurement invariance of the ICT engagement scale was examined across Singapore 
and Turkey. If a sufficient degree of invariance is achieved (i.e., scalar invariance), then the effects of ICT engage- 
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ment factors on performance of science could be compared across these two countries. Specifically, the following 
research questions were addressed: 
1. To what extent does the ICT engagement scale show invariance across Singapore and Turkey? (Multi- 
group CFA) 
2. If a sufficient level of invariance can be established, how do the effects of ICT engagement factors on 
student science performance differ by Singapore and Turkey? (Multi-group SEM) 


Research Methodology 
General Background 


PISA evaluates to what extent 15-year-old students who are about to complete their compulsory education 
have acquired the knowledge and skills necessary for them to adapt to modern societies. This assessment examines 
the extent to which students can draw conclusions from the information they have acquired and to what extent 
they can apply this knowledge in school and out-of-school, rather than whether students can reproduce informa- 
tion. The PISA 2018 evaluation is the seventh since the program started in 1997. In the program, which is carried 
out and developed in cooperation between the governments of OECD countries and partner countries/economies, 
evaluations focusing on basic areas such as reading, mathematics and science are carried out in three-year cycles. 
The main assessment area in PISA 2018 was reading (Mathematics became the main field in 2003 and 2012, and 
science was the main field in 2006 and 2015) (OECD, 2019b). 


Participants 


The data were obtained from the PISA 2018 database which is open access and freely available. Missing values 
were excluded from the analysis. In Turkey 6531 students participated from 189 schools with 3396 being female 
49.3% and 3494 being male 5.7%. In Singapore 6390 students from 166 schools were tested; 3277 were female 
49.1 % and 3399 were male 5.9 %. A large part of the Singapore and Turkey data consisted of grade 10 students 
(respectively 91.3% and 77.8%). 


Instrument and Procedures 


Zylka et al. (2015) introduced ICT engagement scale as a motivation and meta-cognition of ICT literacy. First, 
the ICT engagement scale had a five-dimensional construct, and then it was revised to a four-dimensional construct. 
The revised ICT engagement scale has four factors: interest in ICT, perceived ICT competence, perceived autonomy 
in using ICT and social relatedness in using ICT. ICT engagement scale consists of 4 factors and 21 items in PISA 2018. 

Allitems were measured by 4-point Likert scale. This scale ranged from 1 for strongly disagree to 4 for strongly 
agree (Zylka et al., 2015). Higher values of 4-point scale show better ICT engagement. The means of the items ranged 
from 2.16 to 3.17 and all were higher than the midpoint of the 4-point scale, 2.0. Descriptive statistics and internal 
consistency of the scale (Croanbach’s alpha) are presented in Table 1. Cronbach's alpha coefficients of four factors 
for each of the two countries ranged from .846 to .876 and were sufficiently high. 

Interest in ICT (IICT): IICT defines an individual's long-term choice about the tasks, topics, or activities related 
to ICT. Interest in ICT is expected to influence behavior related to ICT and produce positive emotions, learning and 
performance results (Goldhammer et al., 2016). In PISA 2018, this subscale had six items. Internal consistencies of 
interest in ICT were .792 for the Singapore data and .860 for Turkey data. 

Perceived ICT competence (PICT): PICT is the belief of individuals about their knowledge with respect to ICT 
and to use this knowledge (Goldhammer et al., 2016). This subscale consisted of five items. Internal consistencies 
of perceived ICT competence were .806 for the Singapore data and .872 for Turkey data. 

Perceived autonomy in using ICT (PAICT): PAICT projects the individual’s perceived control and self-manage- 
ment in activities related to ICT (Goldhammer et al., 2016). It is expected to experience a sense of control over by 
using ICT tools and to associate their success in using these tools with their own abilities rather than other reasons 
(Kunina-Habenicht & Goldhammer, 2020). This subscale consisted of five items. Internal consistencies of perceived 
autonomy in using ICT were .867 for the Singapore data and .874 for Turkey data. 

Social relatedness in using ICT (SRICT): It expresses the need for the individual to make ICT a subject in com- 
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munication and interaction with others, and also to share their interest, knowledge, experience and activities with 
others (Goldhammer et al., 2016). This subscale consisted of five items. Internal consistencies of social relatedness 
in using ICT were .850 for the Singapore data and .868 for Turkey data. 


Table 1 
Descriptive Statistics and Reliabilities of ICT Engagement Constructs 


Singapore Turkey Overall 
u(SD) a u(SD) a u(SD) a 
ICT 3.14( 0.68) 792 2.82( 0.44) 860 2.98( 0.56) 846 
PICT 2.95( 0.80) 806 2.80( 0.56) 872 2.87( 0.68) 846 
PAICT 3.02( 0.47) .867 2.72( 0.39) 874 2.87( 0.43) .876 
SRICT 2.69( 0.06) .850 2.72( 0.04) .868 2.70( 0.05) 859 


Science Performance: Science literacy defined within the scope of PISA research is considered as the ability of 
students to deal with issues related to science and to reflect on scientific facts. Science literacy requires the ability 
to explain facts, design research methods, and interpret evaluation data and findings scientifically (OECD, 2019a). 
Science performance in PISA measures students’ science literacy in the use of scientific knowledge to determine 
questions, gain new knowledge, describe scientific phenomena, and draw evidence-based conclusions about is- 
sues related to science. The science performances of 79 countries participating in PISA 2018 were evaluated over 
their average scores and results showed that the average science scores were varied between 59 and 336. Turkey, 
ranked 39 in science among 79 countries participating in PISA 2018, it ranked 30th among the 37 OECD countries 
(OECD, 2019d). Singapore was one of the top countries in science. 

Measurement Invariance: Measurement invariance (MI) evaluates whether a construct is measured and 
interpreted in the same way across different groups and is a requirement for making valid comparisons (Putnick 
& Bornstein, 2016). CFA can be used within the scope of SEM to test measurement invariance. There are four hier- 
archical levels of MI in Multi-group CFA and each of these levels is based on the additional equality constraints on 
the model parameters if the previous invariance is met. Configural invariance, which is the first step of assessing 
measurement invariance, explores if the same items measure the same constructs across groups. Factor models 
belonging to all groups to be compared are estimated at the same time. Since this is the baseline model, the valid- 
ity of the configural invariance is evaluated by the model fit measures. The configural model is a baseline model 
to compare with metric invariance (weak factorial invariance). After providing configural invariance, metric invari- 
ance is explored for whether factor loadings are equivalent across groups. Providing invariance of factor loadings 
shows that the construct has the same meaning across groups. After providing metric invariance, scalar invariance 
of item intercepts, is explored for metric invariant items. Scalar invariance (strong factorial invariance) indicates 
mean equivalence in different groups. After providing scalar invariance, the final step is to test residual invariance. 
The residual invariance (strict invariance) is required for full measurement invariance. The residual invariance is not 
required for testing mean differences. Since the residual invariance is insignificant for the interpretation of latent 
mean differences, this step can be skipped in many studies or if this step cannot be provided, latent factor means 
can be compared (Vandenberg & Lance, 2000). 


Data Analysis 


Mplus 6.1 and R Studio tools were used to analyze the data. First step, exploratory factor analysis (EFA) was 
applied to identify the structure of ICT engagement. After demonstrating the convergent validity and the discrimi- 
nant validity, we used the Cronbach's alpha for assessing factor reliability. 

Before investigating invariance, it is important to ensure that the ICT engagement model provides a good fit 
for both countries. Thus, the second step was to test whether the proposed four-factor ICT engagement construct 
fitted the data from each country using the single group CFA approach. In this step, the four-dimensional construct 
of ICT engagement was tried to confirm for both countries. The measurement models of ICT engagement were 
estimated via CFA using Maximum likelihood estimation (ML). Before conducting the CFA, the assumptions of 
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normality were explored. The skewness values of the items varied between (-.92; -.14) and the kurtosis values of 
items varied between (-.76; .94). If the absolute values of skewness and kurtosis are below 1 deviation from nor- 
mality is defined as slight, between 1 and 2.3, moderate, and above 2.3, severe. (Lei & Lomax, 2005). The absolute 
values of ICT engagement items skewness and kurtosis were below 1 the deviation from normality was weak so ML 
estimation was used in this research (Ooi et al., 2012; Simsek & Tekeli, 2015). To evaluate the measurement model 
fit, three most common fit indices (Comparative Fit Index (CFI), Root Mean Square Error of Approximation (RMSEA) 
and Standardized RMR (SRMR)) were used. CFI values more than .90 and RMSEA and SRMR values smaller than .08 
indicate an acceptable model fit, whereas CFI values more than .95 and RMSEA and SRMR values smaller than .05 
suggest good model fit (Hu & Bentler, 1999; Marsh et al., 2005) 

After the ICT engagement construct was provided for each country, in the third step MI was tested using the 
multistage procedure of Multi-Group CFA. A configural model tested for whether the proposed structure of ICT 
engagement would be equal across the two groups. The configural model included the same number of factors and 
the same number of items. There was not equality restriction on measurement and structural parameters across 
countries. The Singapore and Turkey model were estimated at the same time. After the configural invariance was 
provided, the factor coefficients were kept equal for both groups to test metric invariance. The Likelinood-Ratio 
Test (LR) was used to evaluate the model fit in the measurement invariance analysis (Cheung & Rensvold, 2009). 
The change in CFI (ACFI) criterion which is less sensitive to the sample size and alternative fit indices (ARMSEA, 
ASRMR) were also considered. Chen (2007) suggested that such changes should be equal or less than .01 for the 
ACFI, .015 for the ARMSEA and .010 for the ASRMR. Metric invariance held, then, the scalar invariance model was 
examined. Factor loadings and item intercepts were kept equal in both countries. The results showed that scalar 
invariance model fit was acceptable. Thus, both the factor loadings and the item intercepts of ICT engagement 
scale were invariant across Singapore and Turkey. After scalar invariance held, residual invariance was examined. 
The residual invariance was not supported based on the chi-square difference test. As a result, we demonstrated 
scalar invariance of the scale with 4 factors and 21 items across the two countries; thus, we could proceed to struc- 
tural invariance. When we explored the structural invariance, we only tested the factor mean invariance using the 
method that was recommended by Vandenberg and Lance (2000). Singapore was taken as the reference group 
when comparing factor means, so the estimated factor means for countries were actually the differences in fac- 
tor means between groups. We demonstrated strong factorial invariance including configural invariance, metric 
invariance, and scalar invariance of ICT engagement model across groups, so we were able to use ICT engagement 
to meaningful and valid comparisons. 

For the comparison of the effects of ICT engagement factors on performance of science between these two 
countries, we used the Multi-group SEM analysis in the fourth step. Before comparing the path coefficients across 
the countries, each of the two groups were modeled separately (Jiang et al., 2021). After that, the configural SEM 
model was explored considering the two groups simultaneously without any equality constraint on the structural 
path coefficients. 


Research Results 
Exploratory Factor Analysis (EFA) 


First, EFA was performed on a total of 21 items to determine the factors. Kaiser-Meyer-Olkin (KMO), the mea- 
sure of sampling adequacy should be greater than .5, KMO value was .934 which indicated the suitability for factor 
analysis. Bartlett’s test of sphericity was rejected (p< .001). This demonstrated that there were appropriate correla- 
tions in the data for factor analysis. The EFA results using Principal Components Factoring with Promax rotation 
revealed four significant factors. The results showed that 63.50% of the total variance was explained by these four 
factors. Standardized factor loadings of 21 items are illustrated in Table 2. 

As shown in Table 2, all the item loadings were perfect (Tabachnick & Fiddell, 2007). All the items were loaded 
stronger on the related factors than on the other factors and there were no items with noticeable cross-loads. Thus, 
convergent validity and discriminant validity were demonstrated (Churchill, 1979). The internal consistency of the 
factors was evaluated with Cronbach's alpha and they were found to be above the recommended threshold value 
of .60 (Hair et al., 1998). So, the scale was found to be reliable. 
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Table 2 
Standardized Factor Loadings and Cronbach's a 


Factors lICT PICT PAICT SRICT 
658 702 831 193 
770 811 803 820 
Standardized factor 187 680 178 695 
loadings 737 864 837 862 
128 863 831 820 
818 
Cronbach’s a 846 846 876 859 
Single group CFA 


Table 3 shows the model fit indices of the overall data and each of the country for single group CFA. The re- 
sults showed an acceptable fit for Singapore and the overall data. All the CFI values were greater than .90, all the 
RMSEA and SRMR were less than .08. For Turkey, the results showed a good fit. The CFA was .947 and the RMSEA 
and SRMR were less than .05. All the results showed that the four factor of ICT engagement scale was supported 
in both countries. 


Table 3 
Model Fit Indices of the CFA 


dp) RMSEA (90%-Cl) CFI SRMR 
Overall 15103.396 (400) .075 (.074-.076) 912 079 
Singapore 6967.367 (183) .076 (.075-.078) O11 063 
Turkey 4081.186 (183) .057 (.056-.059) 947 034 


Multi-group CFA 


Table 4 represents the results of the Multi-group CFA. As shown in Table 4, configural model provided accept- 
able fit for the data (CFl=.92>.90, RMSEA=.053<.08 and SRMR=.062<.10) which indicated the construct equivalence 
of the ICT engagement across Singapore and Turkey. The metric model fit was found to be acceptable (CFI=.92>.90, 
RMSEA=.053<.08 and SRMR=.061<.10) and metric invariance was supported based on the chi-square difference 
test (C’,,=2682.127, Adf=17, p>.001). In addition, ACFl<.01, ARMSEA<.015, and ASRMRs.010 (Chen, 2007). So, 
the metric invariance indicated that the items were equivalently related to the latent factors. Furthermore, the 
scalar invariance model fit was found to be acceptable (CFI=.92>.90 RMSEA=.053<.08 and SRMR= .061<.10). We 
also conducted LR test with the current model (Ax? = 16.840; df = 18; p = 5341)c? ,,=1372.716, Adf=17, p>.001) 
Ax? = 10.273; df = 18; p = .9226). and ACFI, ARMSEA and ASRMR indicated that scalar invariance was held. After 
scalar invariance provided, residual invariance was examined. The tests of differences in fit between residual in- 
variance model and scalar model (C g@= BAX =194,690; Adf= 21, p<.001); df = 22; p = .000) indicated that residual 
invariance was not established. 
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Table 4 
Fit Indices for Invariance Tests of Measurement Model 


Model CFI RMSEA — SRMR Ay?(Adf) ACFI ARMSEA ASRMR 
Configural Model 91 075 079 - - - 
Metric Model o 070 068 2682.127(17) 00 -.005 -011 
Scalar Model 92 067 051 1372.716 (17) -01 -.003 -.008 
Residual Model 90 062 071 3194.690 (21) -.02 000 020 


Even if residual invariance cannot be achieved, groups can still be compared on the latent variable; strong 
factorial invariance is sufficient to make valid comparisons. Singapore was considered the reference group where 
all factor means were constrained to equal 0. Actually, estimated factor means were differences in factor means 
between Turkey and Singapore. The factor means invariance demonstrated that the factor means of interest in 
ICT (-.293, p<.001), perceived ICT competence (-.096, p<.001) and perceived autonomy in using ICT (-.326, p<.001) 
were significantly lower in Turkey than in Singapore while the factor mean (.042, p<.001) of social relatedness in 
using ICT was significantly higher in Turkey than in Singapore. 


Multi-group SEM 


We used Multi-group SEM to explore the equality or invariance of the path coefficients across the two countries. 
Before comparing the path coefficients across the countries, each of the two groups were modeled separately. The 
Singapore model was acceptable (RMSEA=.075, 90% Cl=(.074, .077); SRMR =.064; CFl=.900). Interest in ICT (.178, 
p<.001) and perceived autonomy in using ICT (.208, p<.001) had positive effect on science performance. Perceived 
ICT competence (-.055, p<.01) and social interaction using ICT (-.305, p<.001) had negative effect on science per- 
formance. The Turkey model also was acceptable (RMSEA=.06, 90% CI= (.059, .062); SRMR=.037; CFI =.937). In the 
Turkey model, interest in ICT (.205, p<.001), perceived ICT competence (.185, p<.001) and perceived autonomy in 
using ICT (.063, p< .01) had significant positive effect on science performance. Social relatedness in using ICT (-.209, 
p<.001) had negative effect on science performance. Singapore and Turkey baseline SEM model fitted data very well. 

To test the invariance of path coefficients, we estimated a configural SEM model for both Singapore and 
Turkey at the same time. The SEM model of the two groups, which was defined as a baseline model, showed an 
acceptable fit for the data (RMSEA =.062, 90% Cl=(.061, .063); SRMR=.067; CFI = .900). 

As the Multi-group SEM was supported, the direct effect of interest in ICT on science performance was restricted 
to be equal across the groups. The Wald test (c?2=14.654; df=1; p<.001) showed that the direct effect of interest in 
ICT on science performance was different for Singapore and Turkey. The direct effect of interest in ICT on science 
performance was lower in Singapore (.125, p<.001) than in Turkey (.248, p<.001). After that, the direct effect of 
perceived ICT competence on science performance was constrained to be equal across countries. Wald test (c?= 
37.938; df=1; p<.001) indicated that the direct effect of perceived ICT competence on science performance was 
non invariant for Singapore and Turkey. The direct effect of Perceived ICT competence on science performance 
was lower in Singapore (.055, p<.001) than in Turkey (.092, p<.001). Later, the direct effect of perceived autonomy 
in using ICT on science performance was constrained to be equal across the groups. Wald test (c?= 51.826; df=1; 
p<.001) showed that the direct effect of perceived autonomy in using ICT on science performance was non invari- 
ant for Singapore and Turkey. The direct effect of perceived autonomy in using ICT on science performance was 
lower in Singapore (.088, p<.001) than in Turkey (.151, p<.001). After that, the direct effect of social relatedness in 
using ICT on science performance was set to equal across groups. Wald test (c?= 63.334; df=1; p<.001) showed that 
the direct effect of social relatedness in using ICT on science performance was non invariant for Singapore and 
Turkey. The direct effect of social relatedness in using ICT on science performance was lower in Singapore (-.215, 
p<.001) than in Turkey (-.312, p<.001) 
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Discussion 


The aim of this research was to explore the effects of ICT engagement factors on science performance across 
Singapore and Turkey. The first step was to explore the measurement invariance of the ICT engagement scale 
across these two countries. The single group CFA results confirmed the four-dimensional structure of the ICT 
engagement scale. All items were loaded on the related factors of ICT engagement and all factors were reliable. 
The multi-group CFA results demonstrated that ICT engagement scale with 4 factors and 21 items had strong 
factorial invariance (configural, metric and scalar invariance) across these two countries. The ICT engagement 
had the same construct across Singapore and Turkey, so we were able to use ICT engagement to meaningful 
and valid comparisons. After the strong factorial invariance was confirmed, we tested factor mean invariance by 
treating Singapore as the reference group. The factor means invariance model showed that the factor means of 
interest in ICT, perceived ICT competence and perceived autonomy in using ICT were significantly lower in Tur- 
key than in Singapore while the factor mean of social relatedness in using ICT was significantly higher in Turkey 
than in Singapore. According to the International Telecommunication Union (ITU) 2017 report, Singapore with 
a score of 8.05 had a higher ICT performance compared to Turkey with a score of 6.08. Singapore was at the top 
of this ranking, while Turkey was in the middle. Our results were parallel to levels of ICT development for these 
two countries except social relatedness in using ICT. 

The multi group SEM results demonstrated that interest in ICT, perceived ICT competence and perceived 
autonomy in using ICT had positive direct effect on science performance and these effects were lower in Sin- 
gapore than in Turkey. The social relatedness in using ICT had negative direct effect on science performance 
both in Singapore and in Turkey; this effect was lower in Singapore than in Turkey. The positive and significant 
effects of interest in ICT, perceived ICT competence and perceived autonomy in using ICT indicated that both 
Singapore and Turkey considered the role of ICT in education in a similar manner. These results were consistent 
for both countries, and they were compatible with the ICT policies. The negative effects of social relatedness on 
science performance emerged from the possible adverse side effect of ICT. These results were consistent with 
literature as expected. The higher effect for Turkey might be due to the fact that the students use ICT for social 
purposes rather than learning. 

Students’ interest in ICT showed significant positive effect on science performance both in Singapore and 
in Turkey. This result was also supported by previous studies showing that interest in ICT had a positive relation- 
ship with students’ academic performance (Hu et al., 2018; Jansen et al., 2016; Lee & Wu, 2012; Meng et al., 2019; 
Xiao & Hu, 2019). Some of the studies showed that interest in ICT did not have significant effect on academic 
performance (Juhanak et al., 2019), but most of these studies only analyzed a specific country. Students’ interest 
in ICT was related to the pleasure and positive feelings of using products based to ICT such as mobile devices 
or computers (Zylka et al., 2015). According to the results of this research and other literature, students who are 
more interested in ICT use are likely to have better science learning outcomes. This can be explained as students 
who are more interested in ICT will participate in learning activities using computers or the internet more often 
than other students (Jansen et al., 2016; Lee & Wu, 2012). Also, these students will have a more motivated and 
positive attitude towards science learning with technology (Park & Weng, 2020). Students interested in ICT can 
learn science with an interactive educational software, training software or computer tutorial that are designed 
to help student learning. Students interested in ICT learn when and how they can use different tools and teach- 
ers can also inform students to use the right tools. For example, word processing software helps students for 
organizing ideas, writing homework and projects the works. Students can use spreadsheet for analyzing and 
modeling scientific data. 

Students’ perceived ICT competence also showed a significant positive effect on students’ science perfor- 
mance both Singapore and Turkey. This result supported other studies’ results (Hosein et al., 2010; Hu et al., 2018; 
Selwyn & Husen, 2010). Students with high perceived ICT competence can motivate themselves to solve more 
difficult issues with more effective strategies. According to these findings, students with high perceived ICT 
competence were more likely to use software or online resources to work than those with low ICT competencies 
(Hosein et al., 2010). However, some studies supported that ICT competence had no effect or it had negative 
relationship with performance (Juhanak et al.2016; Xiao & Hu, 2019). The results varied depending on students’ 
grade levels and which country was analyzed (Park & Weng, 2020). The competence in using ICT is considered 
an important skill that students need to acquire in order to be successful in the digital age. 
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Students’ perceived ICT autonomy had a significant positive effect on students’ science performance both 
in Singapore and in Turkey. This result was consistent with the literature (Cardenas-Claros & Oyanedel, 2016; 
Xiao & Hu, 2019; Meng et al., 2019). Autonomy implies that students can organize their learning and use ICT to 
complete tasks and achieve mastery (Fu, 2013). Students with higher autonomy in the use of ICT tend to take 
more control of their learning processes with technology. Moreover, when students realize that they can control 
their learning with ICT, they can strengthen their autonomy in using ICT and learn more by using ICT effectively 
(Serhan, 2009; Park & Weng, 2020). Considering the positive relationship between perceived ICT autonomy and 
students’ science performance, it becomes necessary to organize the education system to support autonomy. 
Both parents and teachers can provide support to meet the autonomy needs of students. Teachers can direct 
students to science group work where they will use tools related to ICT and make them aware of their own 
achievements and shortcomings (Park & Weng, 2020). 

Social relatedness in using ICT was significantly a negative effect on science performance both in Singapore 
and in Turkey, and its effect size was larger than ICT interest, competence, and autonomy. These obtained results 
were similar to some previous studies (Paul et al., 2012; Huang, 2018; Hu et al., 2018). For example, Hu et al. (2018) 
reported that students who used ICT for higher social interaction performed lower. Students who use social 
media too much have less time to learn (Englander et al., 2010). While using ICT for learning purposes, students 
may also lose their focus as they interact with other ICT-related activities such as games, chats, and social media 
alerts. Parents and teachers should alert students to the addiction of entertainment as well as motivate them 
to use ICT for learning purposes (Park & Weng, 2020). 


Conclusions and Implications 


In this research the relationship between ICT engagement factors and science performance was explored by 
Multi-group SEM across Turkey and Singapore based on the data for PISA 2018. These countries were selected for 
comparison purpose because they intended to use ICT in education indicated in their national plans. The success 
of Singapore in science performance was evident. So, it was the reference for comparison to the Turkey which 
gained fair improvement in recent PISA cycles. As a preliminary requirement of Multi-group SEM, the measurement 
invariance of ICT scale across these countries was provided hierarchically validating configural, metric, and scalar 
invariance. The Multi-group SEM result indicated that interest, perceived competence, and autonomy in ICT use 
showed a positive effect while social relatedness in using ICT had a negative effect on students’ science perfor- 
mance both Singapore and Turkey. Although these effects were not different in their directions, their magnitudes 
were higher in Turkey than in Singapore. 

This research contributes to the field of education both because it shows that the ICT engagement construct 
is comparable and because it shows the effect of four factors of ICT on science performance separately between 
these two countries. The results of this research can be a guide in educational policies to be determined to increase 
science performance and it will also be useful in positioning ICT in education system. 

The positive effects of interest, perceived competence, and autonomy in ICT use on science performance can 
be regarded as an opportunity to the developing countries like Turkey considering the benefits of ICT in education. 
Turkey was in the middle rank of science performance according to PISA 2018 results, and it might be starting to 
appear in the positive results of these ICT policies implemented. The negative effect of social relatedness in using 
ICT on science performance should be handled by policy makers. 

This research has a few limitations. First, the responses of students from Singapore and Turkey were analyzed. 
The results of this paper are limited to these selected countries. The research can be extended to different countries 
with different cultures and different educational structures. Second, the relationship between ICT engagement 
factors and science performance was explored. For future studies, the effect of ICT engagement factors on reading 
and mathematics performance can be explored. Third, in this research, only the comparison was made according 
to the countries, and no comparison was made by gender. It is planned to explore the measurement invariance of 
the ICT engagement scale according to gender. 
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